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DETAILED ACTION 

EXAMINER'S AMENDMENT 

1. An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 
1 .3 1 2. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview with 
Richard Lyon (Reg. No. 37,385) on 12/16/04. 

The application has been amended as follows: 

In the Claims: 

7. (currently amended) Th e proc e ss of Claim 6, wh e r e in th e fusing n e ural n e twork 
A computer-implemented face recognition process for identifying a person depicted in an input 
image, comprising using a computer to perform the following process actions: 

creating a database of a plurality of model image characterizations, each of which 
represents the face of a known person that it is desired to identify in the input image as well as 
the person's face pose; 

training a neural network ensemble to identify a person and their face pose from a 
region which has been extracted from said input image and characterized in a manner similar to 
the plurality of model images, wherein the network ensemble comprises, a first stage having a 
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plurality of classifiers each of which has input and output units and is dedicated to a particular 
pose range and outputs a measure of the similarity indicative of the similarity between said 
characterized input image region and each of said model image characterizations associated with 
the particular pose range of the classifier, and a fusing neural network as its second stage which 
combines the outputs of the classifiers to generate an output indicative of the person associated 
with the characterized input image region and the face pose of that person and which has at least 
enough output units to allow a different output to represent each person it is desired to identify at 
each of the pose ranges, and wherein th e proc es s action of training th e n e ural n e twork e ns e mbl e 
furth e r compris e s th e actions of: training the neural network ensemble comprises, 

preparing each model image characterization from a model image 
depicting the face of a known person that it is desired to identify in the input image by, 

extracting the portion of the model image depicting said face, 
normalizing the extracted portion of the model image by resizing it 
to a prescribed scale if not already at the prescribed scale and adjusting the region so that the eye 
locations of the depicted subject fall within a prescribed area, and 

cropping the extracted portion of the model image by eliminating 
unneeded portions of the image not specifically depicting part of the face of the subject to create 
a model face image, 

categorizing the model face images by assigning each to one of a set of 
pose ranges into which its associated face pose falls, and 

for each pose range. 
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choosing a prescribed number of the model face images of each 
person being modeled which have been assigned to the selected pose range, 

concatenating each of the chosen model face images to create a 
respective dimensional column vector (DCV) for each, 

computing a covariance matrix from the DCVs, 

calculating eigenvectors and corresponding eigenvalues from the 

covariance matrix, 

ranking the eigenvalues in descending order, 
identifying a prescribed number of the top eigenvalues, 
using the eigenvectors corresponding to the identified eigenvalues 
to form the rows of a basis vector matrix (BVM) for the pose range, and 

multiplying each DCV by each BVM to produce a set of principal 
components analysis (PCA) coefficient vectors for each model face image, and 

for each face recognition neural network, inputting, one at a time, each of 
the PCA coefficient vectors associated with the pose range of the face recognition neural 
network into the inputs of the network until the outputs of the network stabilize^ [;]] 

initializing the fusing neural network for training^;]] 
for each DCV, simultaneously inputting the PCA coefficient vectors 
generated from the DCV into the respective face recognition neural network associated with the 
vector's particular pose range group until all the PCA coefficient vectors of every DCV have 
been input, and repeating until the outputs of the fusing neural network stabilize^;]] and 
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for each DCV 5 simultaneously inputting the PCA coefficient vectors 
generated from the DCV into the respective face recognition neural network associated with the 
vector's particular pose range group and assigning the active output of the fusing neural network 
as corresponding to the particular person and pose associated with the model image used to 
create the set of PCA coefficient vectors ; and 

employing the network ensemble to identify the person associated with the 
characterized input image region and the face pose of that person . 

8. ( pr e viously pr e s e nt e d currently amended ) The process of Claim 7, wherein the 

process action of employing the neural network ensemble to identify the person depicted in the 
input image face region, comprises the actions of: 

preparing the face region extracted from an input image by normalizing and 
cropping the extracted regions, wherein said normalizing comprises resizing the extracted face 
region to the same prescribed scale if not already at the prescribed scale and adjusting the region 
so that the eye locations of the depicted subject fall within a prescribed area, and wherein the 
cropping comprises e liminat e d eliminating unneeded portions of the image not specifically 
depicting part of the face of the subject; 

concatenating the prepared face region to create a DCV; 

multiplying the DCV by each BVM to produce a set of PCA coefficient vectors 
for the extracted face region; 
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inputting each PCA coefficient vector in the set of PCA coefficient vectors into 
the respective face recognition neural network associated with that vector's particular pose range 
group; and 

identifying the active unit of the output of the fusing neural network and 
designating the person and pose previously assigned to that unit as the person and pose 
associated with the extracted face region. 

17. (currently amended) Th e syst e m of Claim 16, wh e r e in th e fusing neural n e twork 
A face recognition system for identifying a person depicted in an input image, comprising: 
a general purpose computing device; and 

a computer program comprising program modules executable by the computing 
device, wherein the computing device is directed by the program modules of the computer 
program to, 

capture model images, each of which depicts at least one person of known 

identity, 

locate and extract regions within the model images, each of which depicts 
the face of a known person that it is desired to identify in the input image, 

determine the face pose for each of the face regions extracted from the 

model images, 

categorize each face region by assigning each to one of a set of pose 
ranges into which its associated face pose falls, 

train a neural network ensemble to identify a person and their face pose 
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from a region that depicts the face of a person which has been extracted from said input image, 
wherein the network ensemble comprises, a first stage having a plurality of classifiers each of 
which has input and output units and is dedicated to a particular pose range and outputs a 
measure of the similarity indicative of the similarity between said input image region and each of 
said model image regions associated with the particular pose range of the classifier, and a fusing 
neural network as its second stage which combines the outputs of the classifiers to generate an 
output indicative of the person associated with the characterized input image region and the face 
pose of that person and which has at least enough output units to allow a different output to 
represent each person it is desired to identify at each of the pose ranges, and wherein th e sub - 
mndnl ft for training th e n e ural n e twork e ns e mbl e farth e r compris e s sub - modul e s for: training the 
neural network ensemble comprises, 

(a) preparing each face region extracted from said model 
images by normalizing and cropping the extracted regions, wherein said normalizing comprises 
resizing each extracted face region to the same prescribed scale if not already at the prescribed 
scale and adjusting each region so that the eve locations of the depicted subject fall within the 
same prescribed area, and wherein said cropping comprises eliminating unneeded portions of the 
image not specifically depicting part of the face of the subject, 

(b) selecting a previously unselected one of the set of pose 

ranges, 

(c) choosing a prescribed number of the prepared face images 
of each person being modeled which have been assigned to the selected pose range. 
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(d) concatenating each of the chosen prepared face images to 
create a respective dimensional column vector (DCV) for each, 

(e) computing a covariance matrix from the DCVs, 

(f) calculating eigenvectors and corresponding eigenvalues 

from the covariance matrix, 

(g) ranking the eigenvalues in descending order, 

(h) identifying a prescribed number of the top eigenvalues, 

(i) using the eigenvectors corresponding to the identified 
eigenvalues to form the rows of a basis vector matrix (BVIVO for the selected pose range, 

(j) repeating actions (fr) through (i) for each remaining pose 

range, 

(k) multiplying each DCV by each BVM to produce a set of 
principal components analysis (PCA) coefficient vectors for each face image, 

(0 for each face recognition neural network, inputting, one at a 
time, each of the PCA coefficient vectors associated with the pose range of the face recognition 
neural network into the inputs of the network until the outputs of the network stabilize.^;]] 

(m) initializing the fusing neural network for training^;]] 

(n) for each DCV, simultaneously inputting the PCA 
coefficient vectors generated from the DCV into the respective face recognition neural network 
associated with the vector's particular pose range group until all the PCA coefficient vectors of 
every DCV have been input, and repeating until the outputs of the fusing neural network 
stabilize.^;]] and 
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(o) for each DCV, simultaneously inputting the PCA 
coefficient vectors generated from the DCV into the respective face recognition neural network 
associated with the vector's particular pose range group and assigning the active output of the 
fusing neural network as corresponding to the particular person and pose associated with the 
model image used to create the set of PCA coefficient vectors , and 

employ the network ensemble to identify the person associated with the 
characterized input image region and their face pose . 

1 8 . ( original currently amended ) The system of Claim 1 7 , wherein the sub - modul e 
module for employing the neural network ensemble to identify the person depicted in the input 
image face region and the pose associated with the face of the identified person, comprises sub- 
modules for: 

preparing the face region extracted from an input image by normalizing and 
cropping the extracted regions, wherein said normalizing comprises resizing the extracted face 
region to the same prescribed scale if not already at the prescribed scale and adjusting the region 
so that the eye locations of the depicted subject fall within a prescribed area, and wherein the 
cropping comprises e liminat e d eliminating unneeded portions of the image not specifically 
depicting part of the face of the subject; 

concatenating the prepared face region to create a DCV; 

multiplying the DCV by each BVM to produce a set of PCA coefficient vectors 
for the extracted face region; 
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inputting each PCA coefficient vector in the set of PCA coefficient vectors into 
the respective face recognition neural network associated with that vector's particular pose range 
group; and 

identifying the active unit of the output of the fusing neural network and 
designating the person and pose previously assigned to that unit as the person and pose 
associated with the extracted face region. 

27. (currently amended) Th e comput e r - r e adabl e m e mory of Claim 26, wh e r e in th e 
fusing n e ural n e twork A computer-readable memory for use in identifying a person depicted in 
an input image, comprising: 

a computer-readable storage medium; and 

a computer program comprising program modules stored in the storage medium, 
wherein the storage medium is so configured by the computer program that it causes a computer 
to, 

input model images, each of which depicts at least one person of known 

identity, 

locate and extract regions within the model images, each of which depicts 
the face of a known person that it is desired to identify in the input image, 

determine the face pose for each of the face regions extracted from the 

model images, 

categorize each face region by assigning each to one of a set of pose 
ranges into which its associated face pose falls. 
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train a neural network ensemble to identify a person and their face pose 
from a region that depicts the face of a person which has been extracted from said input image, 
wherein the network ensemble comprises, a first stage having a plurality of classifiers each of 
which has input and output units and is dedicated to a particular pose range and outputs a 
measure of the similarity indicative of the similarity between said input image region and each of 
said model image regions associated with the particular pose range of the classifier, and a fusing 
neural network as its second stage which combines the outputs of the classifiers to generate an 
output indicative of the person associated with the characterized input image region and the face 
pose of that person and which has at least enough output units to allow a different output to 
represent each person it is desired to identify at each of the pose ranges, and wherein th e sub 
moHnl ft for training th e n e ural n e twork e ns e mbl e furth e r compris e s sub - modul es for: training the 
neural network ensemble comprises, 

(a) preparing each face region extracted from said model 
images by normalizing and cropping the extracted regions, wherein said normalizing comprises 
resizing each extracted face region to the same prescribed scale if not already at the prescribed 
scale and adjusting each region so that the eve locations of the depicted subject fall within the 
same prescribed area, and wherein said cropping comprises eliminating unneeded portions of the 
image not specifically depicting part of the face of the subject, 

(b) selecting a previously unselected one of the set of pose 

ranges, 

(c) choosing a prescribed number of the prepared face images 
of each person being modeled which have been assigned to the selected pose range, 
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(d) concatenating each of the chosen prepared face images to 
create a respective dimensional column vector (DCV) for each, 

(e) computing a covariance matrix from the DCVs, 

(f) calculating eigenvectors and corresponding eigenvalues 

from the covariance matrix , 

(g) ranking the eigenvalues in descending order, 

(h) identifying a prescribed number of the top eigenvalues, 

(i) using the eigenvectors corresponding to the identified 
eigenvalues to form the rows of a basis vector matrix ( BVM) for the selected pose range, 

(j) repeating actions (b) through (i) for each remaining pose 

range, 

(k) multiplying each DCV by each BVM to produce a set of 
principal components analysis (PCA) coefficient vectors for each face image, 

0) for each face recognition neural network, inputting, one at a 
time, each of the PCA coefficient vectors associated with the pose range of the face recognition 
neural network into the inputs of the network until the outputs of the network stabilize^ [;]] 

(m) initializing the fusing neural network for training^;]] 

(n) for each DCV, simultaneously inputting the PCA 
coefficient vectors generated from the DCV into the respective face recognition neural network 
associated with the vector's particular pose range group until all the PCA coefficient vectors of 
every DCV have been input, and repeating until the outputs of the fusing neural network 
stabilize^;]] and 
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(o) for each DCV, simultaneously inputting the PCA 
coefficient vectors generated from the DCV into the respective face recognition neural network 
associated with the vector's particular pose range group and assigning the active output of the 
fusing neural network as corresponding to the particular person and pose associated with the 
model image used to create the set of PCA coefficient vectors , and 

employ the network ensemble to identify the person associated with the 
characterized input image region and their face pose . 

28. ( original currently amended ) The computer-readable memory of Claim 27 
wherein the sub - modul e module for employing the neural network ensemble to identify the 
person depicted in the input image face region and the pose associated with the face of the 
identified person, comprises sub-modules for: 

preparing the face region extracted form an input image by normalizing and 
cropping the extracted regions, wherein said normalizing comprises resizing the extracted face 
region to the same prescribed scale if not already at the prescribed scale and adjusting the region 
so that the eye locations of the depicted subject fall within a prescribed area, and wherein the 
cropping comprises e liminat e d eliminating unneeded portions of the image not specifically 
depicting part of the face of the subject; 

concatenating the prepared face region to create a DCV; 

multiplying the DCV by each BVM to produce a set of PCA coefficient vectors 
for the extracted face region; 



Application/Control Number: 09/536,820 Page 14 

Art Unit: 2623 

inputting each PCA coefficient vector in the set of PCA coefficient vectors into 
the respective face recognition neural network associated with that vector's particular pose range 
group; and 

identifying the active unit of the output of the fusing neural network and 
designating the person and pose previously assigned to that unit as the person and pose 
associated with the extracted face region. 

33. ( pr e viously pr e s e nt e d currently amended ) The process of Claim 32, wherein 
the process action of employing the network ensemble to identify the person depicted in the 
input image face region, comprises the actions of: 

preparing the face region extracted from an input image by normalizing and 
cropping the extracted regions, wherein said normalizing comprises resizing the extracted face 
region to the same prescribed scale if not already at the prescribed scale and adjusting the region 
so that the eye locations of the depicted subject fall within a prescribed area, and wherein the 
cropping comprises e liminat e d eliminating unneeded portions of the image not specifically 
depicting part of the face of the subject; 

concatenating the prepared face region to create a DCV; 

inputting the DCV of the face region into all classifiers; and 

identifying the active output of the neural network and designating the person 
previously assigned to that unit as the person associated with the extracted face 'region. 
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REASONS FOR ALLOWANCE 

2. The following is an examiner's statement of reasons for allowance: the instant invention 
replaces the known face recognition. The essential difference from the prior art is that the instant 
invention uses prepared face image representations to train a neural network ensemble including 
a first stage of classifiers each dedicated to a particular pose range and a second stage in the form 
of a singe fusing neural network. The face images are preprocessed in order to create the 
prepared face image representations including normalizing, cropping, categorizing, and 
abstracting extracted image regions so as to facilitate the comparison process. The abstracting 
procedure represents the images in a simper form to reduce the processing necessary in 
comparison which entails the use of eigenface representations and the creation of PCA 
coefficient vectors to represent each normalized and cropped face image. The eigenface 
approach entails first assigning a prescribed number of the normalized and cropped face images 
associated with each person being modeled to a selected pose range group. Each of these 
assigned face images is then concatenated to create respective DCVs. PCA vectors are generated 
by multiplying a DCV by the BVM associated with the pose range group of the selected face 
recognition neural network. The face recognition neural networks of the first stage of the 
ensemble are trained by inputting, one at a time, each of the PCA coefficient vectors associated 
with the pose range group of a selected face recognition neural network into the inputs of that 
neural network. This is repeated until the outputs of the selected face recognition neural network 
stabilize. The fusing network is initialized for training. The PCA coefficient vectors generated 
from a particular DCV are simultaneously input into the respective face recognition neural 
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network associated with the vector's particular pose range group. This is repeated for each set of 
PCA coefficient vectors. Therefore, the instant invention identifies the person and face pose of 
an image with reduced processing required thereby facilitating the comparison process. These 
features in combination with the other elements of the claims are not disclosed or suggested by 
the prior art of record. 

Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 



CONTACT INFORMATION 

3. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Virginia M Kibler whose telephone number is (703) 306-4072. 
The examiner can normally be reached on Mon-Thurs 8:00 - 5:30 and every other Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Virginia Kibler can be reached on (703) 306-4072. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 




Virginia Kibler 
12/15/04 




